Uncertainty Estimation for Heatmap-based Landmark Localization
基于热图的地标定位的不确定性估计

The recent advancement in machine learning has led to convolutional neural networks (CNNs) dominating the task of landmark localization. Encoder-decoder methods, originally proposed for the task of image segmentation [7], have cemented themselves as one of the leading approaches for landmark localization in both the medical domain [9, 10, 11] and computer vision [12, 13, 14]. The architecture of these methods allow the analysis of images at multiple resolutions, learning to predict a Gaussian heatmap centred around the predicted landmark location. The activation of each pixel in the heatmap can be interpreted as showing the pseudo-probability of that pixel being the target landmark. The network learns to generate a high response near the landmark, smoothly attenuating the responses in a small radius around it. Regressing heatmaps proves more effective than regressing coordinates [15], as the heatmap image offers smoother supervision than direct coordinate values, and also models some uncertainty in the prediction.
机器学习的最新进展导致卷积神经网络（CNN）在地标定位任务中占据主导地位。编码器-解码器方法最初是为图像分割任务而提出的[7]，现已成为医学领域[9,10,11]和计算机视觉[12,13,14]中地标定位的主要方法之一。 ]。这些方法的架构允许以多种分辨率分析图像，学习预测以预测的地标位置为中心的高斯热图。热图中每个像素的激活可以解释为显示该像素作为目标地标的伪概率。网络学会在地标附近生成高响应，平滑地衰减其周围小半径内的响应。事实证明，回归热图比回归坐标更有效[15]，因为热图图像提供比直接坐标值更平滑的监督，并且还对预测中的一些不确定性进行建模。

However, in medical imaging the number of available training samples is often small so the encoder-decoder network is forced to be shallow, compromising its performance [15]. One method to overcome this is via a patch-based approach; alleviating the problem by sampling many small “patches’ from an image, learning the relationship between each patch and the target landmark [16, 17]. This approach can generate orders of magnitude more training samples from a single image compared to the encoder-decoder style methods. Furthermore, patch-based models that use Fully Convolutional Networks (FCN) have fewer parameters than encoder-decoder architectures, decreasing computational requirements and training times [8].
然而，在医学成像中，可用训练样本的数量通常很小，因此编码器-解码器网络被迫变浅，从而损害了其性能[15]。克服这个问题的一种方法是通过基于补丁的方法；通过从图像中采样许多小“补丁”，学习每个补丁与目标地标之间的关系来缓解问题[16, 17]。与编码器-解码器风格的方法相比，这种方法可以从单个图像生成更多数量级的训练样本。此外，使用全卷积网络（FCN）的基于补丁的模型比编码器-解码器架构具有更少的参数，从而减少了计算要求和训练时间[8]。

Noothout et al. [18] implemented a patch-based network using an FCN to jointly perform classification and regression on each patch. The coarse binary classification task determines whether a patch contains the landmark, and the more precise regression task estimates the displacement from the patch to the landmark. This multi-task, joint learning leads to a light-weight network and enhanced localization performance, with the two tasks sharing a feature representation that improves the performance of both [19]. However, the resulting network has a strong local focus and is also susceptible to failure if the predicted patch from the classification task is incorrect. In a follow-up work, Noothout et al. [20] extended their work [18] into a two-stage method: they first train a CNN to provide global estimates for the landmarks, then employ specialised CNNs for each landmark for the final prediction. This method improves upon the first in terms of localization error, but has the drawback of requiring multiple training stages.
努索特等人。 [18]使用FCN实现了基于补丁的网络，对每个补丁联合执行分类和回归。粗略的二元分类任务确定补丁是否包含地标，更精确的回归任务估计从补丁到地标的位移。这种多任务联合学习带来了轻量级网络并增强了定位性能，两个任务共享一个特征表示，从而提高了两者的性能[19]。然而，生成的网络具有很强的局部焦点，如果分类任务的预测补丁不正确，也容易失败。在后续工作中，Noothout 等人。 [20]将他们的工作[18]扩展为两阶段方法：他们首先训练 CNN 来提供地标的全局估计，然后为每个地标使用专门的 CNN 进行最终预测。该方法在定位误差方面改进了第一种方法，但具有需要多个训练阶段的缺点。

To mitigate the inherent local focus of the patch-based methods, we extended the patch-based network [18] by borrowing heatmap regression from the encoder-decoder networks; reforming the binary classification task as a Gaussian heatmap regression task [8]. Named PHD-Net (Patch Heatmap & Displacement), this smoother supervision improved performance, reducing misidentifications compared to using the classification branch from the prior work [18]. Furthermore, we introduced the method Candidate Smoothing, combining the features from the two branches to output more accurate predictions along with an uncertainty measure.
为了减轻基于补丁的方法固有的局部焦点，我们通过借用编码器-解码器网络的热图回归来扩展基于补丁的网络[18]；将二元分类任务改造为高斯热图回归任务[8]。与使用之前工作中的分类分支相比，这种更平滑的监督提高了性能，减少了错误识别，被命名为 PHD-Net（补丁热图和位移）[18]。此外，我们引入了候选平滑方法，结合两个分支的特征来输出更准确的预测以及不确定性度量。

Estimating the uncertainty of machine learning predictions is a topic of growing interest, particularly relevant in the domain of medical imaging where there is often a human in the loop to manually correct flagged predictions. A concentrated effort in uncertainty estimation has been applied to image segmentation by the community, a task similar to landmark localization that instead aims to predict a mask for an entire structure rather than a single point. Some of the most successful approaches use Bayesian approximation methods like Monte-Carlo dropout [21] or an ensemble of networks [22], using the variance of repeated predictions as an indicator for uncertainty. The gold standard approach across many studies is to use an ensemble of networks. This method affords better performance [22] and a more accurate mechanism for Bayesian marginalization [23] compared to a single model using Monte-Carlo dropout. However, the obvious drawback of ensembles is the need to train multiple models.
估计机器学习预测的不确定性是一个越来越令人感兴趣的话题，特别是在医学成像领域，其中经常有人手动纠正标记的预测。社区已将不确定性估计的集中精力应用于图像分割，这是一项类似于地标定位的任务，旨在预测整个结构而不是单个点的掩模。一些最成功的方法使用贝叶斯近似方法，例如蒙特卡洛退出[21]或网络集成[22]，使用重复预测的方差作为不确定性的指标。许多研究的黄金标准方法是使用网络集合。与使用蒙特卡罗 dropout 的单个模型相比，该方法提供了更好的性能 [22] 和更准确的贝叶斯边缘化机制 [23]。然而，集成的明显缺点是需要训练多个模型。

In landmark localization we are predicting a single point rather than a mask, but similar uncertainty estimation approaches can be utilised. However, there are limited works exploring uncertainty in landmark localization. Payer et al. [24] directly modeled uncertainty during training by learning the Gaussian covariances of target heatmaps, and predicting the distribution of likely locations of the landmark at test time. Lee et al. [25] borrowed from image segmentation approaches by proposing a Bayesian CNN that utilised Monte-Carlo dropout to predict the location and subject-level uncertainty of cephalometric landmarks. Another method to measure the subject-level confidence (the inverse of uncertainty) of a heatmap-based landmark prediction is to measure the maximum heatmap activation (MHA) of the predicted heatmap. Since the activation of a Gaussian heatmap at a particular pixel represents the pseudo-probability of the pixel being the landmark, we can use this pseudo-probability as a confidence measure: the higher the activation, the more certain the prediction. Drevicky et al. [6] compared MHA with ensemble and Monte-Carlo dropout methods, finding MHA surprisingly effective given its simplicity. However, similarly to image segmentation, they found using an ensemble of models was best at predicting uncertainty. They calculated the coordinate prediction variance between an ensemble of models, and found this method performed best at estimating prediction uncertainty.
在地标定位中，我们预测单个点而不是掩模，但可以使用类似的不确定性估计方法。然而，探索地标定位不确定性的工作有限。佩耶等人。 [24]通过学习目标热图的高斯协方差，并预测测试时地标的可能位置的分布，直接对训练期间的不确定性进行建模。李等人。 [25]借鉴图像分割方法，提出了贝叶斯 CNN，利用蒙特卡罗 dropout 来预测头影测量地标的位置和主体级不确定性。测量基于热图的地标预测的主体级置信度（不确定性的倒数）的另一种方法是测量预测热图的最大热图激活（MHA）。由于高斯热图在特定像素处的激活表示该像素作为地标的伪概率，因此我们可以使用该伪概率作为置信度度量：激活越高，预测就越确定。德雷维奇等人。 [6] 将 MHA 与集成和蒙特卡罗 dropout 方法进行了比较，发现 MHA 由于其简单性而出奇地有效。然而，与图像分割类似，他们发现使用模型集合最适合预测不确定性。他们计算了模型集合之间的坐标预测方差，并发现该方法在估计预测不确定性方面表现最佳。

In our earlier work utilising the patch-based model PHD-Net, MHA was also used as the uncertainty metric [8]. However, the heatmap analysed is distinctly different from the heatmaps predicted by encoder-decoder networks. Rather than explicitly learning a Gaussian function centred around the landmark, the approach combined patch-wise heatmap and displacement predictions. We produced a new non-Gaussian heatmap, where the activation of each pixel is defined by the number of patches that voted for it, regularised by the coarse global likelihood prediction. Therefore, the resulting heatmap represents patch-wise ensemble votes rather than a Gaussian function, where the MHA is the pixel with the most “patch votes”.
在我们早期使用基于补丁的模型 PHD-Net 的工作中，MHA 也被用作不确定性度量 [8]。然而，分析的热图与编码器-解码器网络预测的热图明显不同。该方法没有明确学习以地标为中心的高斯函数，而是结合了分块热图和位移预测。我们生成了一个新的非高斯热图，其中每个像素的激活由投票给它的补丁数量定义，并通过粗略全局似然预测进行正则化。因此，生成的热图代表补丁方式的集成投票，而不是高斯函数，其中 MHA 是具有最多“补丁投票”的像素。

To the best of our knowledge, no study has investigated how heatmap-based uncertainty estimation measures can be used to filter out poor predictions in landmark localization. Furthermore, no general framework has been proposed to compare how well uncertainty measures can predict localization error - an important practical application in clinical settings.
据我们所知，还没有研究调查如何使用基于热图的不确定性估计措施来过滤地标定位中的不良预测。此外，还没有提出通用框架来比较不确定性测量如何预测定位误差——这是临床环境中的重要实际应用。

First, we briefly review the two models we use for landmark localization, allowing us to compare the generalisability of our uncertainty measures across different heatmap generation approaches. We implement a variation of the popular encoder-decoder networks that regresses Gaussian heatmaps, U-Net [7]. We also implement a patch-based method, PHD-Net [8], which produces a heatmap from patch votes.
首先，我们简要回顾用于地标定位的两个模型，使我们能够比较不同热图生成方法的不确定性度量的通用性。我们实现了一种流行的编码器-解码器网络的变体，它可以回归高斯热图，U-Net [7]。我们还实现了一种基于补丁的方法，PHD-Net [8]，它根据补丁投票生成热图。

Although generated differently, we hypothesise both U-Net and PHD-Net produce heatmaps containing useful information to quantify a prediction’s uncertainty - but are they equally effective? To this end, we compare the performance of both models under three uncertainty estimation methods: a baseline approach, our proposed approach extending the baseline to an ensemble of networks, and the gold standard approach. Each method extracts coordinate values from the predicted heatmap, and estimates the prediction’s uncertainty.
尽管生成方式不同，但我们假设 U-Net 和 PHD-Net 都会生成包含有用信息的热图，以量化预测的不确定性 - 但它们同样有效吗？为此，我们比较了两种模型在三种不确定性估计方法下的性能：基线方法、我们提出的将基线扩展到网络集合的方法以及黄金标准方法。每种方法都从预测的热图中提取坐标值，并估计预测的不确定性。

We leverage the described uncertainty measures to inform the subject-level uncertainty of any given prediction, i.e. is the model’s prediction likely to be accurate, or inaccurate based on this uncertainty value? We propose a data-driven approach, Quantile Binning, using a hold-out validation set to establish thresholds delineating varying levels of uncertainty specific to each trained model. We use these learned thresholds to categorise our predictions into bins and estimate error bounds for each bin. We opt for a data-driven approach compared to a rule-based approach for two reasons: 1) When models are constrained to a limited dataset (in the order of hundreds of training samples), they can have difficulty converging well, particularly in the task of localizing a difficult landmark. 2) If the limited dataset contains high variance, different training sets can lead to confoundingly different learned weights under the same model architecture and training schedule. Therefore, establishing a set of thresholds for each model is more invariant to training differences compared to using the same thresholds for all models.
我们利用所描述的不确定性度量来告知任何给定预测的主体级不确定性，即根据该不确定性值，模型的预测可能是准确的还是不准确的？我们提出了一种数据驱动的方法，即分位数分箱，使用保留验证集来建立阈值，描述每个训练模型特有的不同程度的不确定性。我们使用这些学习到的阈值将我们的预测分类为箱并估计每个箱的误差范围。与基于规则的方法相比，我们选择数据驱动的方法有两个原因：1）当模型被限制在有限的数据集（大约数百个训练样本）时，它们可能很难很好地收敛，特别是在定位困难地标的任务。 2）如果有限的数据集包含高方差，则不同的训练集可能会导致在相同的模型架构和训练计划下产生令人困惑的不同学习权重。因此，与对所有模型使用相同的阈值相比，为每个模型建立一组阈值对于训练差异更具有不变性。

Quantile Binning is application agnostic; applicable to any data as long as it consists of continuous tuples of $(UncertaintyMeasure,\,EvaluationMetric)$.
分位数分箱与应用程序无关；适用于任何数据，只要它由 $(UncertaintyMeasure,EvaluationMetric)(UncertaintyMeasure,\backslash ,EvaluationMetric)(\; italic\_U\; italic\_n\; italic\_c\; italic\_e\; italic\_r\; italic\_t\; italic\_a\; italic\_i\; italic\_n\; italic\_t\; italic\_y\; italic\_M\; italic\_e\; italic\_a\; italic\_s\; italic\_u\; italic\_r\; italic\_e\; ,\; italic\_E\; italic\_v\; italic\_a\; italic\_l\; italic\_u\; italic\_a\; italic\_t\; italic\_i\; italic\_o\; italic\_n\; italic\_M\; italic\_e\; italic\_t\; italic\_r\; italic\_i\; italic\_c\; )$ 的连续元组组成。

In this paper, we generate these pairings after the landmark localization model is trained. We use a hold-out validation set and make coordinate predictions and uncertainty estimates using each of our three uncertainty measures described in Section 4.2. Since we have the ground truth annotations of the validation set we can produce continuous $(UncertaintyMeasure,\,LocalizationError)$ tuples for each uncertainty measure.
在本文中，我们在训练地标定位模型后生成这些配对。我们使用保留验证集，并使用第 4.2 节中描述的三个不确定性度量中的每一个进行坐标预测和不确定性估计。由于我们有验证集的真实注释，我们可以为每个不确定性度量生成连续的 $(UncertaintyMeasure,LocalizationError)(UncertaintyMeasure,\backslash ,LocalizationError)(\; italic\_U\; italic\_n\; italic\_c\; italic\_e\; italic\_r\; italic\_t\; italic\_a\; italic\_i\; italic\_n\; italic\_t\; italic\_y\; italic\_M\; italic\_e\; italic\_a\; italic\_s\; italic\_u\; italic\_r\; italic\_e\; ,\; italic\_L\; italic\_o\; italic\_c\; italic\_a\; italic\_l\; italic\_i\; italic\_z\; italic\_a\; italic\_t\; italic\_i\; italic\_o\; italic\_n\; italic\_E\; italic\_r\; italic\_r\; italic\_o\; italic\_r\; )$ 元组。

In this subsection we construct methods to evaluate how well an uncertainty measure’s predicted bins represent the true error quantiles, and how accurate each bin’s estimated error bounds are.
在本小节中，我们构建方法来评估不确定性度量的预测箱代表真实误差分位数的程度，以及每个箱的估计误差范围的准确度。

We split both datasets into 8 folds, and perform 8-fold cross validation for both U-Net and PHD-Net. For each iteration, we select one fold as our testing set, one our hold-out validation set and the remaining 6 as our training set. We select $T=5$ for the ensemble methods, training 5 separate models at each fold. We chose $T=5$ to compromise with computational constraints, asserting that 5 models are representative to compare the uncertainty methods for our purposes. Each model is identical apart from randomly initialised weights. We randomly select a model for our S-MHA uncertainty measure. For our Quantile Binning method, we select $Q=5$ for 5 bins, balancing the constraints of our small datasets with a useful number of uncertainty categories.
我们将两个数据集分成 8 份，并对 U-Net 和 PHD-Net 进行 8 份交叉验证。对于每次迭代，我们选择 1 个折叠作为我们的测试集，一个作为我们的保留验证集，剩下的 6 个作为我们的训练集。我们选择 $T=5T=5italic\_T\; =\; 5$ 作为集成方法，每次训练 5 个单独的模型。我们选择 $T=5T=5italic\_T\; =\; 5$ 来妥协计算限制，断言 5 个模型具有代表性，可以比较我们的目的的不确定性方法。除了随机初始化的权重之外，每个模型都是相同的。我们为 S-MHA 不确定性测量随机选择一个模型。对于我们的分位数分箱方法，我们选择 $Q=5Q=5italic\_Q\; =\; 5$ 作为 5 个分箱，平衡小数据集的约束与有用数量的不确定性类别。

We implement our U-Net model [7] using the MONAI package [30]. We design the model with 5 encoding-decoding levels, creating 1.63M learnable parameters. We modify the objective function from image segmentation to simultaneous landmark localization, minimising the mean squared error between the target and predicted heatmaps. We use the full $512\times 512$ pixel image as input, and learn heatmaps of the same size. We generate target heatmaps using Eq. (1) with a standard deviation of 8, and train for 1000 epochs with a batch size of 2, and a learning rate of 0.001 using the Adam Optimiser (settings from [8]).
我们使用 MONAI 包 [30] 实现我们的 U-Net 模型 [7]。我们设计了具有 5 个编码-解码级别的模型，创建了 163 万个可学习参数。我们将目标函数从图像分割修改为同步地标定位，从而最小化目标和预测热图之间的均方误差。我们使用完整的 $512\times 512512\backslash times\; 512512\; \times \; 512$ 像素图像作为输入，并学习相同大小的热图。我们使用方程式生成目标热图。 (1) 标准差为 8，使用 Adam 优化器（设置来自 [8]）训练 1000 个周期，批量大小为 2，学习率为 0.001。

We implement our PHD-Net model following [8], creating a model with 0.06M learnable parameters. For all experiments we trained PHD-Net for 1000 epochs using a batch size of 32 and a learning rate of 0.001, using the Adam Optimiser. We train one landmark at a time. Note, the only difference in setup from [8] in this work is different fold splits and training for an additional 500 epochs (same as U-Net) with no early stopping, since we now use our validation set for Quantile Binning.
我们按照 [8] 实现 PHD-Net 模型，创建一个具有 0.06M 可学习参数的模型。对于所有实验，我们使用 Adam 优化器，使用 32 的批量大小和 0.001 的学习率训练 PHD-Net 1000 个时期。我们一次训练一个地标。请注意，这项工作中设置与 [8] 的唯一区别是不同的折叠分割和额外 500 个 epoch 的训练（与 U-Net 相同），没有提前停止，因为我们现在使用分位数分箱的验证集。

Figure 3 and the All rows in Table 1 show the baseline performance for U-Net and PHD-Net at localizing landmarks in our 4CH and SA datasets. We make the following observations:
图 3 和表 1 中的所有行显示了 U-Net 和 PHD-Net 在 4CH 和 SA 数据集中定位地标时的基线性能。我们提出以下观察：

• •

  When considering the entire set of landmarks (All), performance is better on the SA dataset for both models, with PHD-Net outperforming U-Net. On the 4CH dataset, U-Net outperforms PHD-Net, suggesting the higher capacity model of U-Net is more robust to datasets with large variations.

  
  • 当考虑整个地标集（全部）时，两种模型在 SA 数据集上的性能都更好，其中 PHD-Net 的性能优于 U-Net。在 4CH 数据集上，U-Net 的性能优于 PHD-Net，这表明 U-Net 的更高容量模型对于变化较大的数据集更加稳健。
• •

  Simply using a single model with our S-MHA strategy is predictably less robust than ensemble approaches.

  
  • 可以预见的是，简单地使用单个模型与我们的 S-MHA 策略相比，集成方法的鲁棒性较差。
• •

  E-MHA outperforms the previous gold standard of E-CPV for coordinate extraction. However, does it outperform E-CPV in terms of uncertainty estimation? We explore this in Section 6.3.

  
  • E-MHA 优于以前的坐标提取黄金标准E-CPV。然而，它在不确定性估计方面是否优于 E-CPV？我们将在 6.3 节中对此进行探讨。
• •

  The standard deviation in the error for the baseline All results in Table 1 is high for all models. We aspire to catch these bad predictions using quantile binning in Sec. 6.3.

  
  • 对于所有模型，表1 中的所有结果的基线误差的标准偏差都很高。我们渴望在第二节中使用分位数分组来捕捉这些糟糕的预测。 6.3.

We apply quantile binning to each uncertainty measure: S-MHA, E-MHA and E-CPV. We compare results over U-Net and PHD-Net for both the SA and 4CH datasets.
我们将分位数分箱应用于每个不确定性度量：S-MHA、E-MHA 和 E-CPV。我们比较了 SA 和 4CH 数据集的 U-Net 和 PHD-Net 的结果。

We found the most useful information is at the tail ends of the uncertainty distributions. Figs. 3(c) & 3(d) plot the Jaccard Index between ground truth error quantiles and predicted error quantiles. The predictions in the highest uncertainty quantile ($B_{5}$) is significantly better at capturing the correct subset of predictions than the intermediate bins ($B_{2}-B_{4}$). Similarly, in most cases the bin representing the lowest uncertainties ($B_{1}$) had a significantly higher Jaccard Index than the intermediate bins, but still lower than $B_{5}$. Figs. 3(a) & 3(b) show the mean error of the samples of each quantile bin over both datasets. The most significant reduction in localization error is from $B_{5}$ to $B_{4}$ for all uncertainty measures, further indicating our uncertainty measures are well calibrated to filter out gross mispredictions. These findings suggest that most of the utility in the uncertainty measures investigated can be found at the tail ends of the scale. This is an intuitive finding, as the predictions in $B_{5}$ are certainly uncertain, and the predictions in $B_{1}$ are certainly certain.
我们发现最有用的信息位于不确定性分布的尾部。无花果。图 3(c) 和 3(d) 绘制了真实误差分位数和预测误差分位数之间的杰卡德指数。最高不确定性分位数 ( $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ ) 中的预测在捕获正确的预测子集方面明显优于中间箱 ( $B_2-B_{4}B\_\{2\}-B\_\{4\}italic\_B\; start\_POSTSUBSCRIPT\; 2\; end\_POSTSUBSCRIPT\; -\; italic\_B\; start\_POSTSUBSCRIPT\; 4\; end\_POSTSUBSCRIPT$ )。同样，在大多数情况下，代表最低不确定性的 bin ( $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ ) 的杰卡德指数显着高于中间 bin，但仍低于 $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 。无花果。图 3(a) 和 3(b) 显示了两个数据集上每个分位数箱样本的平均误差。对于所有不确定性度量，定位误差最显着的减少是从 $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 到 $B_{4}B\_\{4\}italic\_B\; start\_POSTSUBSCRIPT\; 4\; end\_POSTSUBSCRIPT$ ，进一步表明我们的不确定性度量经过良好校准，可以过滤掉严重的错误预测。这些发现表明，所调查的不确定性度量中的大部分效用可以在量表的末端找到。这是一个直观的发现，因为 $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 中的预测肯定是不确定的，而 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 中的预测肯定是确定的。

The worse trained the landmark localization model, the more useful the uncertainty measure. Table 1 shows the localization error of all methods, models and datasets for the entire set (All) and lowest uncertainty subset ($B_{1}$) of predictions. PHD-Net’s baseline localization performance on the 4CH dataset was worse than U-Net. However, when we consider the lowest uncertainty subset of predictions ($B_{1}$), PHD-Net sees a 47% average reduction in error from all predictions (All), compared to U-Net’s average reduction of 30%. Similarly, U-Net performed worse than PHD-Net for the SA dataset, but saw an average error reduction of 31% compared to PHD-Net’s 25%. This suggests that all investigated uncertainty measures are more effective at identifying gross mispredictions when models are poorly trained.
地标定位模型训练得越差，不确定性度量就越有用。表 1 显示了整个预测集 (All) 和最低不确定性子集 ( $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ ) 的所有方法、模型和数据集的定位误差。 PHD-Net 在 4CH 数据集上的基线定位性能比 U-Net 差。然而，当我们考虑预测的最低不确定性子集 ( $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ ) 时，PHD-Net 发现所有预测 (All) 的误差平均减少了 47%，而 U-Net 的平均误差减少了 30%。同样，U-Net 在 SA 数据集上的表现比 PHD-Net 差，但平均误差比 PHD-Net 的 25% 减少了 31%。这表明，当模型训练不足时，所有研究的不确定性度量在识别严重错误预测方面更有效。

Using heatmap-based uncertainty measures is generalisable across heatmap generation approaches. The bin similarities in Figs. 3(d) & 3(c) show that using S-MHA and E-MHA yields similar performance with PHD-Net and U-Net, despite their different heatmap derivations. Surprisingly using E-MHA does not give a significant increase in bin similarity compared to S-MHA, suggesting the thresholds remain relatively stable across models.
使用基于热图的不确定性度量可以在热图生成方法中推广。图 1 和 2 中的 bin 相似之处图 3(d) 和 3(c) 显示，使用 S-MHA 和 E-MHA 与 PHD-Net 和 U-Net 产生相似的性能，尽管它们的热图推导不同。令人惊讶的是，与 S-MHA 相比，使用 E-MHA 并没有显着增加 bin 相似性，这表明阈值在模型中保持相对稳定。

No investigated method is conclusively best for estimating uncertainty in all scenarios. For the more challenging 4CH data, Fig. 3(c) shows E-CPV is significantly better than S-MHA and E-MHA for both models at capturing the true error quantiles, corroborating the findings of [6]. E-CPV is particularly good at identifying the worst predictions ($B_{5}$). For the easier SA data, no method has a significantly higher Jaccard Index. Therefore, when we generalise across both models and datasets, all uncertainty measures fared broadly similar on average in terms of error reduction between the entire set and the $B_{1}$ subset of predictions. S-MHA had an average error reduction of 35.07%, E-MHA 32.94% and E-CPV 32%.
没有一种研究方法能够最终最好地估计所有情况下的不确定性。对于更具挑战性的 4CH 数据，图 3(c) 显示 E-CPV 在捕获真实误差分位数方面明显优于 S-MHA 和 E-MHA，这证实了 [6] 的发现。 E-CPV 特别擅长识别最坏的预测 ( $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ )。对于更简单的 SA 数据，没有任何方法具有明显更高的 Jaccard 指数。因此，当我们对模型和数据集进行概括时，所有不确定性度量在整个集合和预测的 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 子集之间的误差减少方面平均表现大致相似。 S-MHA 的平均误差降低了 35.07%，E-MHA 降低了 32.94%，E-CPV 降低了 32%。

Despite similar performances in uncertainty estimation, we found E-MHA yields the greatest localization performance overall. Table 1 shows E-MHA offers the best localization performance for $B_{1}$ across both datasets and models. This is due to the combination of offering the most robust coordinate extraction on average (Fig. 3), and similar uncertainty estimation performance (Fig. 3(c), Fig. 3(d)). We more concretely demonstrate Quantile Binning’s ability to identify low uncertainty predictions in Fig. 5. We clearly observe a significant increase in the percentage of images below the acceptable error threshold of 5mm when considering only predictions in $B_{1}$ - with E-MHA giving the greatest proportion of acceptable predictions. For both datasets we observe that PHD-Net using E-MHA has a higher proportion of acceptable predictions compared to U-Net. If we consult Table 1, we also observe that PHD-Net’s predictions in bin $B_{1}$ indeed have lower mean localization error than U-Net’s corresponding bin for both datasets, with E-MHA offering the lowest average localization error.
尽管在不确定性估计方面具有相似的性能，但我们发现 E-MHA 总体上产生了最好的定位性能。表 1 显示 E-MHA 在数据集和模型上为 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 提供了最佳的本地化性能。这是由于提供了平均最稳健的坐标提取（图 3）和类似的不确定性估计性能（图 3(c)、图 3(d)）。我们在图 5 中更具体地证明了分位数分箱识别低不确定性预测的能力。当仅考虑 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 中的预测时，我们清楚地观察到低于 5mm 可接受误差阈值的图像百分比显着增加 - E-MHA 给出了可接受的预测的最大比例。对于这两个数据集，我们观察到与 U-Net 相比，使用 E-MHA 的 PHD-Net 具有更高比例的可接受预测。如果我们查阅表 1，我们还观察到，对于两个数据集，PHD-Net 在 bin $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 中的预测确实比 U-Net 的相应 bin 具有更低的平均定位误差，而 E-MHA 提供了最低的平均定位误差。

We analyse how accurate the isotonically regressed estimated error bounds are for our quantile bins. Figs. 3(e) & 3(f) show the percentage of samples in each bin that fall between the estimated error bounds.
我们分析了分位数箱的等渗回归估计误差范围的准确度。无花果。图 3(e) 和 3(f) 显示每个箱中落在估计误差范围内的样本百分比。

We found we can predict the error bounds for the two extreme bins better than the intermediate bins. Figs. 3(e) & 3(f) show a similar pattern to the Jaccard Index Figs. 3(c) & 3(d), with the two extreme bins $B_{5}$ and $B_{1}$ predicting error bounds significantly more accurately than the inner bins. Again, this indicates the most useful uncertainty information is present at the extremes of the uncertainty distribution, with the predicted uncertainty-error function unable to capture a consistent relationship for the inner quantiles. This finding is intuitive, as it is easier to put a lower error bound on the most uncertain quantile of predictions or upper bound on the most certain predictions, than it is to assign tighter error bounds on middling uncertainty values.
我们发现我们可以比中间箱更好地预测两个极端箱的误差范围。无花果。图 3(e) 和 3(f) 显示了与 Jaccard 索引图类似的模式。 3(c) 和 3(d)，两个极端 bin $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 和 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 预测误差范围的准确率明显高于内部 bin。同样，这表明最有用的不确定性信息存在于不确定性分布的极端处，而预测的不确定性误差函数无法捕获内部分位数的一致关系。这一发现很直观，因为与对中等不确定性值指定更严格的误差界限相比，对最不确定的预测分位数设置误差下限或对最确定的预测设置上限更容易。

We also found that a well defined upper bound for heatmap activations is important for error bound estimates. For both the 4CH and SA datasets, S-MHA for PHD-Net is significantly more accurate at predicting error bounds for the highest uncertainty quantile $B_{5}$ compared to the lowest uncertainty quantile $B_{1}$ (56% & 72% compared to 30% & 27% for 4CH & SA, respectively), correlating with S-MHA capturing a greater proportion of those bins (Jaccard Indexes of 32% & 24% compared to 16% & 15%). On the other hand, U-Net using S-MHA predicts error bounds for low uncertainty bins better than high uncertainty bins. This suggests that although PHD-Net’s heatmap activation is a robust predictor of gross mispredictions, the less tight upper bound of its heatmap activations make it hard to make an accurate prediction for the lowest uncertainty quantile ($B_{1}$). This is alleviated by using an ensemble of networks in E-MHA, where the $B_{1}$ bound accuracy is improved to 62%.
我们还发现，明确定义的热图激活上限对于误差界限估计很重要。对于 4CH 和 SA 数据集，与最低不确定性分位数 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 相比，PHD-Net 的 S-MHA 在预测最高不确定性分位数 $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 的误差范围方面显着更准确（56% 和 72% 与 30% 相比） 4CH 和 SA 分别为 27% 和 27%），与 S-MHA 捕获这些垃圾箱的更大比例相关（杰卡德指数分别为 32% 和 24%，而 16% 和 15%）。另一方面，使用 S-MHA 的 U-Net 可以比高不确定性箱更好地预测低不确定性箱的误差范围。这表明，尽管 PHD-Net 的热图激活是严重错误预测的稳健预测器，但其热图激活的上限不太严格，因此很难对最低不确定性分位数 ( $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ ) 做出准确预测。通过在 E-MHA 中使用网络集合可以缓解这一问题，其中 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 边界精度提高到 62%。

E-MHA and E-CPV are more consistent than S-MHA. Overall, there is no significant difference between the error bound estimation accuracy of E-MHA and S-MHA, but Figs. 3(e) & 3(f) show E-MHA has less variation in performance between U-Net and PHD-Net compared to S-MHA, suggesting an ensemble of models is more robust. For the 4CH dataset, PHD-Net using E-CPV is on average significantly more accurate at predicting error bounds than S-MHA and E-MHA. However, there are no significant differences for PHD-Net on the easier SA dataset, nor U-Net on either dataset. There are also no significant differences between U-Net and PHD-Net in error bound estimation accuracy, with each method broadly equally effective for both models.
E-MHA 和 E-CPV 比 S-MHA 更加一致。总体而言，E-MHA 和 S-MHA 的误差界估计精度没有显着差异，但图图 3(e) 和 3(f) 显示，与 S-MHA 相比，E-MHA 在 U-Net 和 PHD-Net 之间的性能差异较小，这表明模型集合更加稳健。对于 4CH 数据集，使用 E-CPV 的 PHD-Net 在预测误差范围方面平均比 S-MHA 和 E-MHA 更准确。然而，PHD-Net 在更简单的 SA 数据集上没有显着差异，U-Net 在任一数据集上也没有显着差异。 U-Net 和 PHD-Net 之间的误差界限估计精度也没有显着差异，每种方法对于这两种模型大致同样有效。

This paper presented a general framework to assess any continuous uncertainty measure in landmark localization, demonstrating its use on three uncertainty metrics and two paradigms of landmark localization model. We introduced a new coordinate extraction and uncertainty estimation method, E-MHA, offering the best baseline localization performance and competitive uncertainty estimation.
本文提出了一个评估地标定位中任何连续不确定性度量的通用框架，展示了其在三个不确定性度量和地标定位模型的两种范式上的使用。我们引入了一种新的坐标提取和不确定性估计方法 E-MHA，提供最佳的基线定位性能和有竞争力的不确定性估计。

Our experiments indicate that both heatmap-based uncertainty metrics (S-MHA, E-MHA), as well as the gold standard coordinate variance uncertainty metric (E-CPV) are applicable to both U-Net and PHD-Net. Despite the two models’ distinctly different approaches to generating heatmaps, using the maximum heatmap activation as an indicator for uncertainty is effective for both models. We showed that all investigated uncertainty metrics were effective at filtering out the gross mispredictions ($B_{5}$) and identifying the 20% most certain predictions ($B_{1}$), but could not capture useful information for the intermediate uncertainty bins ($B_{2}$-$B_{4}$).
我们的实验表明，基于热图的不确定性度量（S-MHA、E-MHA）以及黄金标准坐标方差不确定性度量（E-CPV）都适用于 U-Net 和 PHD-Net。尽管这两个模型生成热图的方法截然不同，但使用最大热图激活作为不确定性的指标对于这两个模型都是有效的。我们表明，所有研究的不确定性指标都能有效过滤掉严重的错误预测 ( $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ ) 并识别 20% 最确定的预测 ( $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ )，但无法捕获中间不确定性区间的有用信息 ( $B_{2}B\_\{2\}italic\_B\; start\_POSTSUBSCRIPT\; 2\; end\_POSTSUBSCRIPT$ - $B_{4}B\_\{4\}italic\_B\; start\_POSTSUBSCRIPT\; 4\; end\_POSTSUBSCRIPT$ ）。

Our experiments also showed that E-MHA and S-MHA had a surprisingly similar ability to capture the true error quantiles of the best and worst 20% of predictions (Figs. 3(c) & 3(d)), but E-MHA was more consistent with its performance predicting the error bounds of those bins across models (Figs. 3(e) & 3(f)). This suggests that the calibration of the head and tail ends of the heatmap distributions is stable across our ensemble of models, but susceptible to variance when fitting our isotonically regressed line to predict error bounds. On the more challenging 4CH dataset, E-CPV broadly remained the gold standard for filtering out the worst predictions, but this trend did not continue in the easier SA dataset (Fig 5).
我们的实验还表明，E-MHA 和 S-MHA 在捕获最佳和最差 20% 预测的真实误差分位数方面具有惊人的相似能力（图 3(c) 和 3(d)），但 E-MHA与其跨模型预测这些箱的误差范围的性能更加一致（图 3(e) 和 3(f)）。这表明热图分布的头部和尾部的校准在我们的模型集合中是稳定的，但在拟合我们的等渗回归线以预测误差范围时容易受到方差的影响。在更具挑战性的 4CH 数据集上，E-CPV 基本上仍然是过滤掉最差预测的黄金标准，但这种趋势在更简单的 SA 数据集中并没有延续（图 5）。

In terms of error bound estimation, we found a strong correlation with the bin’s Jaccard Index to the true quantiles. Bins $B_{5}$ and $B_{1}$ could offer good error bound estimates, but the intermediate bins could not (Figs. 3(e) & 3(f)). We found all uncertainty methods performed broadly the same: effective at predicting error bounds for $B_{1}$ and $B_{5}$, but poor at predicting error bounds for $B_{2}$-$B_{4}$. The one exception was PHD-Net using S-MHA, which could not accurately predict error bounds for $B_{1}$ due to the high variance in pixel activations of highly certain predictions.
在误差界限估计方面，我们发现箱的杰卡德指数与真实分位数有很强的相关性。 bin $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 和 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 可以提供良好的误差范围估计，但中间 bin 不能（图 3(e) 和 3(f)）。我们发现所有不确定性方法的表现大致相同：在预测 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 和 $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 的误差范围方面有效，但在预测 $B_{2}B\_\{2\}italic\_B\; start\_POSTSUBSCRIPT\; 2\; end\_POSTSUBSCRIPT$ - $B_{4}B\_\{4\}italic\_B\; start\_POSTSUBSCRIPT\; 4\; end\_POSTSUBSCRIPT$ 的误差范围方面较差。一个例外是使用 S-MHA 的 PHD-Net，由于高度确定的预测的像素激活的高方差，它无法准确预测 $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 的误差范围。

Overall, we found that using E-MHA provided the most robust coordinate extraction method of all methods, showing the best baseline localization error. When considering only the predictions with the lowest uncertainty ($B_{1}$), using E-MHA achieves the lowest mean localization error accross all models and datasets.
总的来说，我们发现使用 E-MHA 提供了所有方法中最稳健的坐标提取方法，显示出最佳的基线定位误差。当仅考虑不确定性最低的预测 ( $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ ) 时，使用 E-MHA 可在所有模型和数据集中实现最低的平均定位误差。

In terms of utility, when resources are available to train an ensemble of models, we recommend to use E-MHA as the coordinate extraction and uncertainty estimation method. E-MHA offers the best baseline localization performance with a sufficient ability to filter out the gross mispredictions ($B_{5}$) and identify the most certain predictions ($B_{1}$). It can also sufficiently estimate error bounds for these two bins. However, between these thresholds the uncertainty metric is poorly calibrated to bin predictions in a robust way, or accurately predict error bounds. If resources are constrained, S-MHA is surprisingly effective at capturing the true error quantiles for bins $B_{1}$ and $B_{5}$, but note that when using a patch-based voting heatmap that is not strictly bounded, the error bound estimation for $B_{1}$ is not robust.
就实用性而言，当有资源可用于训练模型集合时，我们建议使用 E-MHA 作为坐标提取和不确定性估计方法。 E-MHA 提供了最佳的基线定位性能，有足够的能力过滤掉严重的错误预测 ( $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ ) 并识别最确定的预测 ( $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ )。它还可以充分估计这两个容器的误差范围。然而，在这些阈值之间，不确定性度量很难以稳健的方式校准到箱预测，或者准确地预测误差范围。如果资源有限，S-MHA 在捕获 bin $B_{1}B\_\{1\}italic\_B\; start\_POSTSUBSCRIPT\; 1\; end\_POSTSUBSCRIPT$ 和 $B_{5}B\_\{5\}italic\_B\; start\_POSTSUBSCRIPT\; 5\; end\_POSTSUBSCRIPT$ 的真实误差分位数方面出人意料地有效，但请注意，当使用不严格限制的基于补丁的投票热图时，🚠 的误差界限估计4🚠不健壮。

Beyond the above recommendations, we hope the framework described in this paper can be used to assess refined or novel uncertainty metrics for landmark localization, and act as a baseline for future work. Furthermore, we have shown that both the voting derived heatmap of PHD-Net, and the regressed Gaussian heatmap of U-Net can be exploited for uncertainty estimation. In this paper, we only explored the activation of the peak pixel, but it is likely that more informative measures can be extracted from the broader structure of the heatmap, promising greater potential for uncertainty estimation in landmark localization waiting to be uncovered.
除了上述建议之外，我们希望本文描述的框架可用于评估地标定位的精细或新颖的不确定性指标，并作为未来工作的基线。此外，我们还表明，PHD-Net 的投票导出热图和 U-Net 的回归高斯热图都可以用于不确定性估计。在本文中，我们仅探讨了峰值像素的激活，但很可能可以从更广泛的热图结构中提取更多信息量，从而有望在等待被发现的地标定位中进行不确定性估计的更大潜力。